Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

Test large vector mean operator and fix a few bugs #16079

Merged
merged 8 commits into from
Sep 5, 2019

Conversation

apeforest
Copy link
Contributor

@apeforest apeforest commented Sep 3, 2019

Description

  • Add test for mean operator with large vector.
  • Fixed a few bugs that may affect other large array operators.
  • Fixed a few tests in test_large_vector to make it more memory efficient.

#pragma unroll
for (int i = ndim-1, j = mdim, s = 1; i >= 0; --i) {
Copy link
Contributor

@access2rohit access2rohit Sep 3, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice catch ! :)

@@ -168,6 +168,12 @@ def test_topk():
val = nd.topk(b, k=1, axis=0, dtype=np.int64, ret_typ="value")
assert val.sum() == (LARGE_X - 1)


def test_mean():
a = nd.arange(-LARGE_X / 2, LARGE_X / 2 + 1, dtype=np.int64)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: even though LARGE_X is divisible by 2 but can we use integer division here LARGE_X//2

Copy link
Contributor

@ChaiBapchya ChaiBapchya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

def test_clip():
a = create_vector(LARGE_X)
res = nd.clip(a, a_min=100, a_max=1000)
assert np.sum(res[-1].asnumpy() == 1000) == 1
assert res[-1] == 1000
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good ... previous one was bad

@access2rohit
Copy link
Contributor

Nice work !
Removal of all numpy functions should speedup the processes

@apeforest
Copy link
Contributor Author

@access2rohit My test test_mean runs through. However, test_sequence_last seems to run out of memory on an instance with 480G memory.

test_large_vector.test_slice ... ok
test_large_vector.test_ndarray_zeros ... ok
test_large_vector.test_ndarray_ones ... ok
test_large_vector.test_ndarray_random_uniform ... ok
test_large_vector.test_ndarray_random_randint ... ok
test_large_vector.test_ndarray_empty ... ok
test_large_vector.test_elementwise ... ok
test_large_vector.test_clip ... ok
test_large_vector.test_argmin ... ok
test_large_vector.test_take ... ok
test_large_vector.test_slice_assign ... ok
test_large_vector.test_expand_dims ... ok
test_large_vector.test_squeeze ... ok
test_large_vector.test_broadcast_div ... ok
test_large_vector.test_Dense ... ok
test_large_vector.test_argsort ... ok
test_large_vector.test_sort ... ok
test_large_vector.test_topk ... ok
test_large_vector.test_mean ... ok
test_large_vector.test_ndarray_random_exponential ... ok
test_large_vector.test_ndarray_random_gamma ... ok
test_large_vector.test_ndarray_random_generalized_negative_binomial ... ok
test_large_vector.test_ndarray_random_multinomial ... ok
test_large_vector.test_ndarray_random_negative_binomial ... ok
test_large_vector.test_ndarray_random_normal ... ok
test_large_vector.test_ndarray_random_poisson ... ok
test_large_vector.test_ndarray_random_randn ... ok
test_large_vector.test_ndarray_random_shuffle ... ok
test_large_vector.test_exponent_logarithm_operators ... ok
test_large_vector.test_power_operators ... ok
test_large_vector.test_sequence_mask ... ok
test_large_vector.test_sequence_reverse ... ok
test_large_vector.test_sequence_last ...
Segmentation fault: 11

Segmentation fault: 11

Stack trace:
[bt] (0) /home/ubuntu/src/mxnet/python/mxnet/../../build/libmxnet.so(+0x8f96a9) [0x7fc345d596a9]
[bt] (1) /lib/x86_64-linux-gnu/libc.so.6(+0x354b0) [0x7fc366cc94b0]
[bt] (2) /home/ubuntu/src/mxnet/python/mxnet/../../build/libmxnet.so(+0x1909334) [0x7fc346d69334]
[bt] (3) /usr/lib/x86_64-linux-gnu/libgomp.so.1(+0xf43e) [0x7fc352a0543e]
[bt] (4) /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7fc3670656ba]
[bt] (5) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fc366d9b41d]
Stack trace:
[bt] (0) /home/ubuntu/src/mxnet/python/mxnet/../../build/libmxnet.so(+0x8f96a9) [0x7fc345d596a9]
[bt] (1) /lib/x86_64-linux-gnu/libc.so.6(+0x354b0) [0x7fc366cc94b0]
[bt] (2) /home/ubuntu/src/mxnet/python/mxnet/../../build/libmxnet.so(+0x1909334) [0x7fc346d69334]
[bt] (3) /usr/lib/x86_64-linux-gnu/libgomp.so.1(GOMP_parallel+0x3f) [0x7fc352a01cbf]
[bt] (4) /home/ubuntu/src/mxnet/python/mxnet/../../build/libmxnet.so(mxnet::op::SequenceLastOp<mshadow::cpu, float, float>::Forward(mxnet::OpContext const&, std::vector<mxnet::TBlob, std::allocatormxnet::TBlob > const&, std::vector<mxnet::OpReqType, std::allocatormxnet::OpReqType > const&, std::vector<mxnet::TBlob, std::allocatormxnet::TBlob > const&, std::vector<mxnet::TBlob, std::allocatormxnet::TBlob > const&)+0x50b) [0x7fc346d8838b]
[bt] (5) /home/ubuntu/src/mxnet/python/mxnet/../../build/libmxnet.so(mxnet::op::OperatorState::Forward(mxnet::OpContext const&, std::vector<mxnet::TBlob, std::allocatormxnet::TBlob > const&, std::vector<mxnet::OpReqType, std::allocatormxnet::OpReqType > const&, std::vector<mxnet::TBlob, std::allocatormxnet::TBlob > const&)+0xab0) [0x7fc345eca990]
[bt] (6) /home/ubuntu/src/mxnet/python/mxnet/../../build/libmxnet.so(mxnet::imperative::PushOperator(mxnet::OpStatePtr const&, nnvm::Op const*, nnvm::NodeAttrs const&, mxnet::Context const&, std::vector<mxnet::engine::Var*, std::allocatormxnet::engine::Var* > const&, std::vector<mxnet::engine::Var*, std::allocatormxnet::engine::Var* > const&, std::vector<mxnet::Resource, std::allocatormxnet::Resource > const&, std::vector<mxnet::NDArray*, std::allocatormxnet::NDArray* > const&, std::vector<mxnet::NDArray*, std::allocatormxnet::NDArray* > const&, std::vector<unsigned int, std::allocator > const&, std::vector<mxnet::OpReqType, std::allocatormxnet::OpReqType > const&, mxnet::DispatchMode)::{lambda(mxnet::RunContext, mxnet::engine::CallbackOnComplete)#3}::operator()(mxnet::RunContext, mxnet::engine::CallbackOnComplete) const+0x2aa) [0x7fc345d5241a]
[bt] (7) /home/ubuntu/src/mxnet/python/mxnet/../../build/libmxnet.so(std::_Function_handler<void (mxnet::RunContext), mxnet::imperative::PushOperator(mxnet::OpStatePtr const&, nnvm::Op const*, nnvm::NodeAttrs const&, mxnet::Context const&, std::vector<mxnet::engine::Var*, std::allocatormxnet::engine::Var* > const&, std::vector<mxnet::engine::Var*, std::allocatormxnet::engine::Var* > const&, std::vector<mxnet::Resource, std::allocatormxnet::Resource > const&, std::vector<mxnet::NDArray*, std::allocatormxnet::NDArray* > const&, std::vector<mxnet::NDArray*, std::allocatormxnet::NDArray* > const&, std::vector<unsigned int, std::allocator > const&, std::vector<mxnet::OpReqType, std::allocatormxnet::OpReqType > const&, mxnet::DispatchMode)::{lambda(mxnet::RunContext)#4}>::_M_invoke(std::_Any_data const&, mxnet::RunContext&&)+0x1d) [0x7fc345d533ed]
[bt] (8) /home/ubuntu/src/mxnet/python/mxnet/../../build/libmxnet.so(+0x84c3de) [0x7fc345cac3de]

@apeforest
Copy link
Contributor Author

@wkcn Can you help to review and merge this for me?

@apeforest apeforest added the pr-awaiting-merge Review and CI is complete. Ready to Merge label Sep 5, 2019
Copy link
Member

@wkcn wkcn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the update!

In the unittest test_ndarray_random_randint,
it will be specific to use 2 ** 32 rather than 4294967296.

tests/nightly/test_large_vector.py Outdated Show resolved Hide resolved
tests/nightly/test_large_vector.py Outdated Show resolved Hide resolved
Copy link
Member

@wkcn wkcn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I will merge it after the CI pass.

@wkcn wkcn merged commit d0fa8c0 into apache:master Sep 5, 2019
@apeforest apeforest deleted the bugfix/large-array-mean branch September 5, 2019 09:13
gyshi pushed a commit to gyshi/incubator-mxnet that referenced this pull request Sep 7, 2019
* add test_mean and fix a few potential bugs

* address reviewer commnet

* fix and refactor test

* fix topk test

* address reviewer comment
gyshi pushed a commit to gyshi/incubator-mxnet that referenced this pull request Sep 7, 2019
* add test_mean and fix a few potential bugs

* address reviewer commnet

* fix and refactor test

* fix topk test

* address reviewer comment
access2rohit pushed a commit to access2rohit/incubator-mxnet that referenced this pull request Sep 25, 2019
* add test_mean and fix a few potential bugs

* address reviewer commnet

* fix and refactor test

* fix topk test

* address reviewer comment
access2rohit pushed a commit to access2rohit/incubator-mxnet that referenced this pull request Sep 25, 2019
* add test_mean and fix a few potential bugs

* address reviewer commnet

* fix and refactor test

* fix topk test

* address reviewer comment
access2rohit pushed a commit to access2rohit/incubator-mxnet that referenced this pull request Sep 25, 2019
* add test_mean and fix a few potential bugs

* address reviewer commnet

* fix and refactor test

* fix topk test

* address reviewer comment
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
pr-awaiting-merge Review and CI is complete. Ready to Merge
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants